Test adaptation system

ABSTRACT

Systems and methods are provided for determining a first test automate is associated with the same application as a second test automate, and for each action in the first test automate, determining a common action in the second test automate, determining that an element identifier in the first test automate for the common action has changed in the second test automate, and recording the changed element identifier in a data store, the data store comprising a list of changed element identifiers for each of a plurality of applications. The data store is provided so that element identifiers in at least one custom test automate corresponding to the changed element identifiers for at least one application are automatically determined based on the list of changed element identifiers for each of the plurality of applications.

BACKGROUND

When initially releasing a software product and for subsequent releasesof the software product, a developer of the software product may alsodevelop test automates (e.g., test scripts) for consumers of thesoftware product to use to test various functionality of the softwareproduct. For example, a software product may include an application forcreating a sales order. The developer may develop a test automate thatsteps through functions of the application (e.g., the various functionsto create a sales order) similar to how an end user may use theapplication. The test automate may then be used by a consumer computingdevice or system to test the functionality in the application beforereleasing the application to end users.

BRIEF DESCRIPTION OF THE DRAWINGS

Various ones of the appended drawings merely illustrate exampleembodiments of the present disclosure and should not be considered aslimiting its scope.

FIG. 1 is a block diagram illustrating a networked system, according tosome example embodiments.

FIG. 2 illustrates a portion of an example test automate, according tosome example embodiments.

FIG. 3 illustrates an XML version of the example test automate of FIG.2, according to some example embodiments.

FIG. 4 is a flow chart illustrating aspects of a method, according tosome example embodiments.

FIG. 5 is a flow chart illustrating aspects of a method, according tosome example embodiments.

FIG. 6 is a block diagram illustrating an example of a softwarearchitecture that may be installed on a machine, according to someexample embodiments.

FIG. 7 illustrates a diagrammatic representation of a machine, in theform of a computer system, within which a set of instructions may beexecuted for causing the machine to perform any one or more of themethodologies discussed herein, according to an example embodiment.

DETAILED DESCRIPTION

Systems and methods described herein relate to a test adaptation systemfor updating custom test automates based on changes made to standardtest automates deployed with a software product (e.g., comprising one ormore applications). As explained above, when releasing a softwareproduct and subsequent releases, a developer of the software product mayalso develop a plurality of standard test automates (e.g., test scripts)for consumers of the software products to use to test variousfunctionality of the software product. For example, a software productmay include a first application for creating a sales order. Thedeveloper may develop a standard test automate that steps throughfunctions of the application (e.g., the various functions to create asales order) similar to how an end user may use the application. Thestandard test automate for creating the sales order application may thenbe used by a consumer computing device to test the functionality in theapplication before releasing the application to end users. The consumerof the software product may also develop custom automates based on thestandard automates provided by the developer. For example, the consumermay determine that its end users perform a certain action or use acertain functionality of an application differently than what is coveredin the standard automate provided by the developer. Accordingly, theconsumer may create one or more custom automates, based on one or morestandard automates, for one or more applications with additional ordifferent functionality to test out the functionality or actions thatare specific to their users and use cases. Significant time is spentdeveloping such custom automates and a consumer may create a largenumber of custom automates.

For each release of the software product (e.g., a new version includingchanges to one or more aspects of one or more of the applications of thesoftware product), the software product developer may update one or morestandard automates. Each standard automate is associated with anapplication identifier and comprises a set of actions. Each actioncomprises at least one element identifier (e.g., a string of charactersused to uniquely identify an element in an automate). When a standardautomate is updated, new element identifiers are generated for eachaction that is updated or modified. Accordingly, the previous elementidentifiers are no longer valid.

Thus, when a consumer receives a new release of the software product,and a consumer computing device uses the custom automates to test outthe new release, the custom automates produce a number of errors whichmake the software product or particular application appear to theconsumer as if it is not working or working incorrectly. This is becausethe custom automates are using the previous element identifiers that areno longer valid. This increases the number of incidents raised by theconsumer to the developer incorrectly raising incidents for testfailures (assuming features are not working correctly), significantlydelays release of the updated software product to end users, causessignificant effort on the consumer side to try and replicate changesmade by the developer into numerous custom automates, reduces productivetest time, and the like. Moreover, a list of changes made to thesoftware product and/or the standard automates may be provided in adocument hundreds of pages long, which makes it impossible to manuallydetermine specific relevant changes in any practical timeframe.

Example embodiments address such issues by analyzing updated standardautomates before deployment to a consumer to determine changes toelement identifiers in each standard automate. These changes are storedin a data store and associated with an application identifier. The datastore is deployed to the consumer with the updated standard automatesand updated software product. When the consumer computing device runsthe custom automates, the changed element identifiers in the customautomates can be automatically updated based on the changes in the datastore to ensure all the element identifiers are valid. Any applicationidentifiers may also be updated if needed. Further details of exampleembodiments are described below.

FIG. 1 is a block diagram illustrating a networked system 100, accordingto some example embodiments. The system 100 may include one or moreclient devices such as client device 110. The client device 110 maycomprise, but is not limited to, a mobile phone, desktop computer,laptop, portable digital assistants (PDA), smart phone, tablet,ultrabook, netbook, laptop, multi-processor system, microprocessor-basedor programmable consumer electronic, game console, set-top box, computerin a vehicle, or any other communication device that a user may utilizeto access the networked system 100. In some embodiments, the clientdevice 110 may comprise a display module (not shown) to displayinformation (e.g., in the form of user interfaces). In furtherembodiments, the client device 110 may comprise one or more of touchscreens, accelerometers, gyroscopes, cameras, microphones, globalpositioning system (GPS) devices, and so forth. The client device 110may be a device of a user 106 that is used to access and utilize cloudservices, among other applications.

One or more users 106 may be a person, a machine, or other means ofinteracting with the client device 110. In example embodiments, the user106 may not be part of the system 100 but may interact with the system100 via the client device 110 or other means. For instance, the user 106may provide input (e.g., touch screen input or alphanumeric input) tothe client device 110 and the input may be communicated to otherentities in the system 100 (e.g., third-party servers 130, server system102, etc.) via the network 104. In this instance, the other entities inthe system 100, in response to receiving the input from the user 106,may communicate information to the client device 110 via the network 104to be presented to the user 106. In this way, the user 106 may interactwith the various entities in the system 100 using the client device 110.

The system 100 may further include a network 104. One or more portionsof network 104 may be an ad hoc network, an intranet, an extranet, avirtual private network (VPN), a local area network (LAN), a wirelessLAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), ametropolitan area network (MAN), a portion of the Internet, a portion ofthe public switched telephone network (PSTN), a cellular telephonenetwork, a wireless network, a WiFi network, a WiMax network, anothertype of network, or a combination of two or more such networks.

The client device 110 may access the various data and applicationsprovided by other entities in the system 100 via web client 112 (e.g., abrowser, such as the Internet Explorer® browser developed by Microsoft®Corporation of Redmond, Wash. State) or one or more client applications114. The client device 110 may include one or more client applications114 (also referred to as “apps”) such as, but not limited to, a webbrowser, a search engine, a messaging application, an electronic mail(email) application, an e-commerce site application, a mapping orlocation application, an enterprise resource planning (ERP) application,a customer relationship management (CRM) application, an analyticsdesign application, and the like.

In some embodiments, one or more client applications 114 may be includedin a given client device 110, and configured to locally provide the userinterface and at least some of the functionalities, with the clientapplication(s) 114 configured to communicate with other entities in thesystem 100 (e.g., third-party servers 130, server system 102, etc.), onan as-needed basis, for data and/or processing capabilities not locallyavailable (e.g., access location information, access software versioninformation, access an ERP system, access a CRM system, access ananalytics design system, access data to respond to a search query, toauthenticate a user 106, to verify a method of payment, access testdata, access one or more standard automates or related data, etc.).Conversely, one or more applications 114 may not be included in theclient device 110, and then the client device 110 may use its webbrowser to access the one or more applications hosted on other entitiesin the system 100 (e.g., third-party servers 130, server system 102,etc.).

A server system 102 may provide server-side functionality via thenetwork 104 (e.g., the Internet or wide area network (WAN)) to one ormore third-party servers 130 and/or one or more client devices 110. Theserver system 102 may include an application program interface (API)server 120, a web server 122, and test adaptation system 124 that may becommunicatively coupled with one or more databases 126.

The one or more databases 126 may be storage devices that store datarelated to users of the system 100, applications associated with thesystem 100, cloud services, and so forth. The one or more databases 126may further store information related to third-party servers 130,third-party applications 132, client devices 110, client applications114, users 106, and so forth. In one example, the one or more databases126 may be cloud-based storage.

The server system 102 may be a cloud computing environment, according tosome example embodiments. The server system 102, and any serversassociated with the server system 102, may be associated with acloud-based application, in one example embodiment.

The test adaptation system 124 may provide back-end support forthird-party applications 132 and client applications 114, which mayinclude cloud-based applications. The test adaptation system 124processes updated test automates to generate element identifier changedata, and so forth, as described in further detail below. The testadaptation system 124 may comprise one or more servers or othercomputing devices or systems.

The system 100 may further include one or more third-party servers 130.The one or more third-party servers 130 may include one or morethird-party application(s) 132. The one or more third-partyapplication(s) 132, executing on third-party server(s) 130, may interactwith the server system 102 via API server 120 via a programmaticinterface provided by the API server 120. For example, one or more thethird-party applications 132 may request and utilize information fromthe server system 102 via the API server 120 to support one or morefeatures or functions on a website hosted by the third party or anapplication hosted by the third party. The third-party website orapplication 132, for example, may provide analytics design and viewingfunctionality that is supported by relevant functionality and data inthe server system 102.

In one example, actions for an application are recorded into a document(e.g., XML or another format). For example, actions that would beperformed by a user to complete a sales order in a sales orderapplication are recorded and stored in a document. Then a testing tool(e.g., Tricentis' Tosca, HP's UFT, or other testing tool), is used toread the document and generate a test automate (e.g., an automatedversion of the recorded actions in the document). Accordingly, a testautomate comprises each of the recorded actions. The test automate alsoincludes at least one element and associated element identifier for eachaction.

FIG. 2 is a display 200 of a portion of an example test automate for anapplication for creating a sales order. The display 200 includes anaction identifier (ID) 202 for each of a plurality of actions, anoptional action 204, an action type 206, a label 208, a value 210, databinding 212, and an option to edit details of technical details 214.FIG. 3 is an XML version of the example test automate in FIG. 2. Thesefigures are discussed in further detail below.

FIG. 4 is a flow chart illustrating aspects of a method 400 forgenerating a data store of changed element identifiers to update one ormore custom automates, according to some example embodiments. Forillustrative purposes, method 400 is described with respect to thenetworked system 100 of FIG. 1. It is to be understood that method 400may be practiced with other system configurations in other embodiments.

As explained above, a significant number of standard test automates maybe generated for a plurality of applications associated with a softwareproduct. The relevant standard test automates of the plurality ofstandard test scrips may be updated based on a new release includingchanges or new functionality to the plurality of applications. Forexample, if a change in layout or functionality is made to anapplication to create a sales order, the standard test automate for thatapplication may also be changed to test the new functionality, removeany testing for old functionality, and so forth. To determine whatchanges are made to the standard test automates, each test automate iscompared to a previous version of the test automate to determine what,if any, changes are made.

In operation 402, for each of a plurality of standard test automates, acomputing system (e.g., server system 102 or test adaptation system 124)determines whether a first standard test automate is associated with thesame application as a second standard test automate. For example, thecomputing system may access a data store (e.g., one or more database(s)126 and/or other data store) that contains new versions of testautomates (e.g., corresponding to the new release of one or moreapplications of the software product) and previous versions of the testautomates (e.g., corresponding to the previous release of one or moreapplications of the software product). The computing system will want tobe sure the first standard test automate and the second standard testautomate are for the same application, and thus, are different versionsof the same standard test automate. In one example a version number foreach test automate is maintained.

In one example, the application is a web page. For example, theapplication is a web page for creating a sales order. For instance, auser can enter various information needed for the sales order, such asan order type, a supplier name, a purchasing organization, a companycode, and so forth. In one example, the computing system determineswhether the first test automate is associated with the same applicationas the second test automate by comparing an application identifier ofthe first test automate with the application of the second test automateto confirm they are the same. In the example where the application is aweb page, the application identifier is a Uniform Resource Locator (URL)or a Uniform Resource Identifier (URI) or other unique identifier forthe application. In the example XML shown in FIG. 3, the applicationidentifier 302 is within the <LABEL> tags, e.g., “PurchaseOrder-create.”

In another example, the computing system determines whether the firsttest automate is associated with the same application as the second testautomate based on a similarity score associated with actions in commonbetween the first test script and the second test script. For example,the computing system compares the application identifier associated withthe first test automate to the application identifier of the second testautomate to determine whether the application identifier associated withthe first test automate and the application identifier of the secondtest automate are the same. Using the example of where the applicationis a web page, the computing system compares the URL, URI, or otherunique application identifier of the first test automate to the URL,URI, or other unique application identifier of the second test automate.For example, the application identifier 302 associated with a first testautomate shown in FIG. 3 is compared to an application identifier in asecond test automate.

In one example, if the computing system determines that the applicationidentifiers are the same, the computing system determines that theapplication is the same and does not proceed further. In anotherexample, even though the computing system determines that theapplication identifiers are the same, the computing device stillproceeds to ensure (e.g., double-check) that the application is thesame. Thus, in the latter example, the computing system determineswhether a first test automate is associated with a same application as asecond test automate based on an application identifier associated withthe first test automate and an application identifier associated withthe second test automate and based on a similarity of actions in commonbetween the first test automate and the second test automate.

If the computing system determines that the application identifierassociated with the first test automate is not the same as theapplication identifier of the second test automate (or isdouble-checking that the applications are indeed the same), thecomputing device compares the actions of the first test automate withthe actions of the second test automate to determine a similarity score.The computing system may use one or more of several factors to determinethe similarity score. For example, the computing system may use thetotal number of actions (e.g., count the number of actions, such as 11actions shown in FIG. 2) in each test automate, may compare each actionin the first test automate to each action in the second test automate,and so forth.

In one example, the similarity score indicates a percentage ofsimilarity between the actions in the first test automate and theactions of the second test automate. For instance, a similarity scoremay be 0.45 to indicate that 45% of the actions (or other aspects of thetest automates) are similar. In one example, a similarity threshold isset to determine whether the first test automate and the second testautomate are for the same application. For example, the similaritythreshold may be 0.75 and thus, if a similarity score is greater than orequal to the similarity threshold, the computing system determines thatthe first test automate and the second test automate are for the sameapplication. Accordingly, the computing system compares the similarityscore to the similarity threshold to determine whether the similarityscore meets the similarity threshold. The computing device determinesthat the first test automate is associated with a same application as asecond test automate based on determining that the similarity scoremeets the similarity threshold (e.g., the similarity score is equal toor greater than 0.75). The computing system determines that the firsttest automate is not associated with a same application as a second testautomate based on determining that the similarity score does not meetthe similarity threshold (e.g., the similarity score is less than 0.75).

If the computing system determines that the application identifierassociated with the first test automate is not the same as theapplication identifier of the second test automate but that the firsttest automate is associated with a same application as a second testautomate based on determining that the similarity score meets thesimilarity threshold, the computing system can store the changedapplication identifier in a data store comprising a list of changedapplication identifiers.

Once the computing system determines that the first test automate isassociated with the same application as the second test automate, foreach action in the first test automate, the computing system determinesa common action in the second test automate, as shown in operation 404.For example, the computing system compares each action in the first testautomate to each action in the second test automate to determine acommon action. In one example, the computing system starts with thefirst action in the first test automate and compares it to each of theactions in the second test automate until it determines a common actionin the second test script. In one example, the computing system comparesan action type and/or an action identifier of the first action in thefirst test automate with an action type and/or an action identifier ofeach action in the second test automate, and so forth. If an action typeand action identifier are the same, the actions are considered the same.

For each common action, the computing system determines whether anelement identifier in the first test automate for the common action haschanged in the second test automate, as shown in operation 406. Forexample, the computing system compares one or more element identifiersin the common action in the first test automate to one or more elementidentifiers in the second test automate to see if there has been achange in one or more element identifiers. Using the example XML of afirst test automate in FIG. 3, the computing system compares elementidentifier 306 M0:46:1:1:2::0:0 of common action 304 of the first testautomate to an element identifier of the common action of the secondtest automate. If an element identifier has not changed, the computingsystem returns to operation 404 until all the actions of the first testautomate have been analyzed.

If an element identifier has changed, as determined in operation 406,the computing system records the changed element identifier(s) in a datastore, as shown in operation 408. In one example, the data storecomprises a list of changed element identifiers (e.g., the previouselement identifier and the new element identifier) for each of aplurality of applications. The data store may be an object in JSONformal, CSV formal, XML format or other format that can be provided to aconsumer computing device or stored in one or more databases (e.g.,database(s) 126) to be accessed by a consumer computing device or othersystem. As explained above, the computing system can also store (e.g.,record) any changed application identifiers in a data store comprising alist of changed application identifiers, similar to how the changedelement identifiers are stored.

In operation 410, the computing system provides the data store (e.g.,either directly or via access to a database or other data store) to aconsumer computing system or computing device to adapt custom automatesthat the consumer has generated. The element identifiers in at least onecustom test automate corresponding to the changed element identifiersfor at least one application can be automatically determined based onthe list of changed element identifiers for each of the plurality ofapplications. Any changed application identifiers can also beautomatically determined based on the list of changed applicationidentifiers for each of the plurality of applications.

Once the consumer computing system receives the data store from thecomputing system, the consumer computing system can adapt customautomates (e.g., automatically determine element identifiers in at leastone custom test automate corresponding to the changed elementidentifiers for at least one application based on the list of changedelement identifiers for each of the plurality of applications). FIG. 5is a flow chart illustrating aspects of a method 500 for adapting customautomates based on changed element identifiers (and/or changeapplication identifiers), according to some example embodiments. Forillustrative purposes, method 500 is described with respect to thenetworked system 100 of FIG. 1. It is to be understood that method 500may be practiced with other system configurations in other embodiments.

In operation 502, the consumer computing system determines, for eachcustom test automate, an application identifier associated with anapplication for the custom test automate. As explained above, in oneexample, the application is a web page, and the application identifieris a Uniform Resource Locator (URL) or a Uniform Resource Identifier(URI) or other unique identifier for the application. In other examples,the application may be in a different format and have another form ofunique identifier. In the example of FIG. 3, the XML format for the testautomate 300 may be for a custom test automate and the applicationidentifier 302 may be “PurchaseOrder-create.”

Once the consumer computing system determines the application identifierfor the custom test automate, the consumer computing system determineselement identifiers in the custom test automate, in operation 504 (e.g.,such as M0:46:1:1:2::0:0, as described above). For example, the consumercomputing system searches through the custom test automate to identifyeach element identifier in the custom test automate.

In operation 506, the consumer computing device determines whether anyelement identifiers in the custom test automate associated with theapplication identifier have been changed. For example, the consumercomputing device accesses the data store of changed element identifiersto determine, for the application identifier, whether any of the elementidentifiers in the custom test automate are associated with changedelement identifiers.

If the consumer computing device determines that one or more elementidentifiers in the customer test automate are associated with changedelement identifiers, the consumer computing device updates the elementidentifiers in the custom test automate that have been changed with thenew element identifiers, in operation 508. For example, the consumercomputing device edits the test automate to replace the previous elementidentifiers with the changed element identifiers.

In one example, the consumer computing device automatically makes thechanges to replace the previous element identifiers with the changedelement identifiers. In another example, the consumer computing devicedisplays the changes via a graphical user interface (GUI) and asks forapproval to make one or more of the changes. A user may accept or rejectany of the changes. Upon receiving approval (e.g., via a userinteraction with the GUI) for one or more of the changes, the consumercomputing device automatically makes the changes replace the previouselement identifiers with the changed element identifiers.

In one example, the consumer computing device determines that theapplication identifier does not correspond to an application identifierin the data store. As explained above, it is possible that theapplication identifier has changed in the new standard automates. Thus,the consumer computing device can check a list of changed applicationidentifiers to see whether the application identifier for the customtest automate has changed. If the application identifier for the customtest automate has changed, the consumer computing device updates theapplication identifier (e.g., edits the custom test automate to replacethe previous application identifier with the new identifier) and thenthe consumer computing device can use the new application identifier todetermine whether any element identifiers in the custom test automatehave been changed (e.g., using operations 504-508 described above).

In another example, there may be no list of changed applicationidentifiers, or the application identifier of the custom test automatemay not be included in the list of changed application identifiers. Inthis case, after determining the application identifier associated withthe application for a custom test automate does not correspond to anapplication identifier in the data store, the consumer computing systemcompares the actions of the custom test automate with the actions ofeach of a plurality of standard test automates in the data score todetermine a similarity score, similar to what is described above. Forexample, the consumer computing system determines whether the similarityscore meets a similarity threshold. The consumer computing devicedetermines that the custom test automate is associated with a sameapplication as a standard test automate based on determining that thesimilarity score meets the similarity threshold. The consumer computingdevice determines that the custom test automate is not associated with asame application as a standard test automate based on determining thatthe similarity score does not meet the similarity threshold.

In the example described with respect to FIG. 5, the consumer computingsystem is performing all the operations. In another example, thecomputing system on a backend system (e.g., server system 102, testadaptation system 124, etc.) could instead perform some or all theoperations in FIG. 5.

The following examples describe various embodiments of methods,machine-readable media, and systems (e.g., machines, devices, or otherapparatus) discussed herein.

Example 1

A computer-implemented method comprising:

for each of a plurality of test automates, determining whether a firsttest automate is associated with a same application as a second testautomate based on an application identifier associated with the firsttest automate and an application identifier associated with the secondtest automate, or based on a similarity score associated with actions incommon between the first test automate and the second test automate;

based on determining that the first test automate is associated with thesame application as the second test automate, for each action in thefirst test automate:

-   -   determining a common action in the second test automate;    -   determining whether an element identifier in the first test        automate for the common action has changed in the second test        automate; and    -   based on determining that an element identifier in the first        test automate for the common action has changed in the second        test automate, recording the changed element identifier in a        data store, the data store comprising a list of changed element        identifiers for each of a plurality of applications; and

providing the data store comprising the list of changed elementidentifiers for each of the plurality of applications to a computingdevice, wherein element identifiers in at least one custom test automatecorresponding to the changed element identifiers for at least oneapplication are automatically determined based on the list of changedelement identifiers for each of the plurality of applications.

Example 2

A method according to any of the previous examples, wherein determiningwhether the first test automate is associated with the same applicationas the second test automate based on the application identifierassociated with the first test automate and the application identifierassociated with the second test automate comprises comparing theapplication identifier associated with the first test automate to theapplication identifier of the second test automate to confirm that theyare the same.

Example 3

A method according to any of the previous examples, wherein theapplication identifier associated with the first test automate or thesecond test automate is a Uniform Resource Locator or a Uniform ResourceIdentifier.

Example 4

A method according to any of the previous examples, wherein determiningwhether the first test automate is associated with the same applicationas the second test automate based on a similarity of actions in commonbetween the first test automate and the second test automate comprises:

comparing the application identifier associated with the first testautomate to the application identifier of the second test automate;

determining that the application identifier associated with the firsttest automate is not the same as the application identifier of thesecond test automate;

comparing the actions of the first test automate with the actions of thesecond test automate to determine the similarity score;

determining whether the similarity score meets a similarity threshold;and

determining that the first test automate is associated with the sameapplication as the second test automate based on determining that thesimilarity score meets the similarity threshold.

Example 5

A method according to any of the previous examples, wherein thesimilarity score is based on a percentage of similarity between theactions of the first test automate and the actions of the second testautomate.

Example 6

A method according to any of the previous examples, further comprising:

recording the application identifier of the second automate in the datastore as a changed application identifier.

Example 7

A method according to any of the previous examples, wherein determiningwhether the first test automate is associated with the same applicationas the second test automate is based on an application identifierassociated with the first test automate and an application identifierassociated with the second test automate and based on a similarity ofactions in common between the first test automate and the second testautomate.

Example 8

A method according to any of the previous examples, whereinautomatically determining element identifiers in at least one customtest automate corresponding to the changed element identifiers for atleast one application based on the list of changed element identifiersfor each of the plurality of applications comprises:

for each custom test automate, determining an application identifierassociated with an application for the custom test automate;

determining element identifiers in the custom test automate;

accessing the data store to determine whether any element identifiers inthe custom test automate associated with the application identifier havebeen changed; and

updating the element identifiers in the custom test automate that havebeen changed with new element identifiers identified in the data store.

Example 9

A method according to any of the previous examples, further comprising:

after determining an application identifier associated with anapplication for a third test automate, determining that the applicationidentifier does not correspond to an application identifier in the datastore;

comparing the actions of the third test automate with the actions ofeach of a plurality of standard test automates in the data score todetermine a similarity score;

determining whether the similarity score meets a similarity threshold;and

determining that the third test automate is associated with a sameapplication as a fourth test automate based on determining that thesimilarity score meets the similarity threshold.

Example 10

A system comprising:

a memory that stores actions; and

one or more processors configured by the actions to perform operationscomprising:

-   -   for each of a plurality of test automates, determining whether a        first test automate is associated with a same application as a        second test automate based on an application identifier        associated with the first test automate and an application        identifier associated with the second test automate, or based on        a similarity score associated with actions in common between the        first test automate and the second test automate;    -   based on determining that the first test automate is associated        with the same application as the second test automate, for each        action in the first test automate:        -   determining a common action in the second test automate;        -   determining whether an element identifier in the first test            automate for the common action has changed in the second            test automate; and        -   based on determining that an element identifier in the first            test automate for the common action has changed in the            second test automate, recording the changed element            identifier in a data store, the data store comprising a list            of changed element identifiers for each of a plurality of            applications; and    -   providing the data store comprising the list of changed element        identifiers for each of the plurality of applications to a        computing device, wherein element identifiers in at least one        custom test automate corresponding to the changed element        identifiers for at least one application are automatically        determined based on the list of changed element identifiers for        each of the plurality of applications.

Example 11

The system of claim 11, wherein determining whether the first testautomate is associated with the same application as the second testautomate based on the application identifier associated with the firsttest automate and the application identifier associated with the secondtest automate comprises comparing the application identifier associatedwith the first test automate to the application identifier of the secondtest automate to confirm they are the same.

Example 12

A system according to any of the previous examples, wherein theapplication identifier associated with the first test automate or thesecond test automate is a Uniform Resource Locator or a Uniform ResourceIdentifier.

Example 13

A system according to any of the previous examples, wherein determiningwhether the first test automate is associated with the same applicationas the second test automate based on a similarity of actions in commonbetween the first test automate and the second test automate comprises:

comparing the application identifier associated with the first testautomate to the application identifier of the second test automate;

determining that the application identifier associated with the firsttest automate is not the same as the application identifier of thesecond test automate;

comparing the actions of the first test automate with the actions of thesecond test automate to determine the similarity score;

determining whether the similarity score meets a similarity threshold;and

determining that the first test automate is associated with a sameapplication as a second test automate based on determining that thesimilarity score meets the similarity threshold.

Example 14

A system according to any of the previous examples, wherein thesimilarity score is based on a percentage of similarity between theactions of the first test automate and the actions of the second testautomate.

Example 15

A system according to any of the previous examples, the operationsfurther comprising:

recording the application identifier of the second automate in the datastore as a changed application identifier.

Example 16

A system according to any of the previous examples, wherein determiningwhether a first test automate is associated with the same application asthe second test automate is based on an application identifierassociated with the first test automate and an application identifierassociated with the second test automate and based on a similarity ofactions in common between the first test automate and the second testautomate.

Example 17

A system according to any of the previous examples, whereinautomatically determining element identifiers in at least one customtest automate corresponding to the changed element identifiers for atleast one application based on the list of changed element identifiersfor each of the plurality of applications comprises:

for each custom test automate, determining an application identifierassociated with an application for the custom test automate;

determining element identifiers in the custom test automate;

accessing the data store to determine whether any element identifiers inthe custom test automate associated with the application identifier havebeen changed; and

updating the element identifiers in the custom test automate that havebeen changed with new element identifiers identified in the data store.

Example 18

A system according to any of the previous examples, further comprising:

after determining an application identifier associated with anapplication for a third test automate, determining that the applicationidentifier does not correspond to an application identifier in the datastore;

comparing the actions of the third test automate with the actions ofeach of a plurality of standard test automates in the data score todetermine a similarity score;

determining whether the similarity score meets a similarity threshold;and

determining that the third test automate is associated with a sameapplication as a fourth test automate based on determining that thesimilarity score meets the similarity threshold.

Example 19

A non-transitory computer-readable medium comprising actions storedthereon that are executable by at least one processor to cause acomputing device to perform operations comprising:

for each of a plurality of test automates, determining whether a firsttest automate is associated with a same application as a second testautomate based on an application identifier associated with the firsttest automate and an application identifier associated with the secondtest automate, or based on a similarity score associated with actions incommon between the first test automate and the second test automate;

based on determining that the first test automate is associated with thesame application as the second test automate, for each action in thefirst test automate:

-   -   determining a common action in the second test automate;    -   determining whether an element identifier in the first test        automate for the common action has changed in the second test        automate; and    -   based on determining that an element identifier in the first        test automate for the common action has changed in the second        test automate, recording the changed element identifier in a        data store, the data store comprising a list of changed element        identifiers for each of a plurality of applications; and

providing the data store comprising the list of changed elementidentifiers for each of the plurality of applications to a computingdevice, wherein element identifiers in at least one custom test automatecorresponding to the changed element identifiers for at least oneapplication are automatically determined based on the list of changedelement identifiers for each of the plurality of applications.

Example 20

A non-transitory computer-readable medium according to any of theprevious examples, wherein determining whether the first test automateis associated with the same application as the second test automatebased on a similarity of actions in common between the first testautomate and the second test automate comprises:

comparing the application identifier associated with the first testautomate to the application identifier of the second test automate;

determining that the application identifier associated with the firsttest automate is not the same as the application identifier of thesecond test automate;

comparing the actions of the first test automate with the actions of thesecond test automate to determine the similarity score;

determining whether the similarity score meets a similarity threshold;and

determining that the first test automate is associated with a sameapplication as the second test automate based on determining that thesimilarity score meets the similarity threshold.

FIG. 6 is a block diagram 600 illustrating software architecture 602,which can be installed on any one or more of the devices describedabove. For example, in various embodiments, client devices 110 andservers and systems 130, 102, 120, 122, and 124 may be implemented usingsome or all of the elements of software architecture 602. FIG. 6 ismerely a non-limiting example of a software architecture, and it will beappreciated that many other architectures can be implemented tofacilitate the functionality described herein. In various embodiments,the software architecture 602 is implemented by hardware such as machine700 of FIG. 7 that includes processors 710, memory 730, and I/Ocomponents 750. In this example, the software architecture 602 can beconceptualized as a stack of layers where each layer may provide aparticular functionality. For example, the software architecture 602includes layers such as an operating system 604, libraries 606,frameworks 608, and applications 610. Operationally, the applications610 invoke application programming interface (API) calls 612 through thesoftware stack and receive messages 614 in response to the API calls612, consistent with some embodiments.

In various implementations, the operating system 604 manages hardwareresources and provides common services. The operating system 604includes, for example, a kernel 620, services 622, and drivers 624. Thekernel 620 acts as an abstraction layer between the hardware and theother software layers, consistent with some embodiments. For example,the kernel 620 provides memory management, processor management (e.g.,scheduling), component management, networking, and security settings,among other functionality. The services 622 can provide other commonservices for the other software layers. The drivers 624 are responsiblefor controlling or interfacing with the underlying hardware, accordingto some embodiments. For instance, the drivers 624 can include displaydrivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers,flash memory drivers, serial communication drivers (e.g., UniversalSerial Bus (USB) drivers), WI-FI® drivers, audio drivers, powermanagement drivers, and so forth.

In some embodiments, the libraries 606 provide a low-level commoninfrastructure utilized by the applications 610. The libraries 606 caninclude system libraries 630 (e.g., C standard library) that can providefunctions such as memory allocation functions, string manipulationfunctions, mathematic functions, and the like. In addition, thelibraries 606 can include API libraries 632 such as media libraries(e.g., libraries to support presentation and manipulation of variousmedia formats such as Moving Picture Experts Group-4 (MPEG4), AdvancedVideo Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3),Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec,Joint Photographic Experts Group (JPEG or JPG), or Portable NetworkGraphics (PNG)), graphics libraries (e.g., an OpenGL framework used torender in two dimensions (2D) and in three dimensions (3D) graphiccontent on a display), database libraries (e.g., SQLite to providevarious relational database functions), web libraries (e.g., WebKit toprovide web browsing functionality), and the like. The libraries 606 canalso include a wide variety of other libraries 634 to provide many otherAPIs to the applications 610.

The frameworks 608 provide a high-level common infrastructure that canbe utilized by the applications 610, according to some embodiments. Forexample, the frameworks 608 provide various graphic user interface (GUI)functions, high-level resource management, high-level location services,and so forth. The frameworks 608 can provide a broad spectrum of otherAPIs that can be utilized by the applications 610, some of which may bespecific to a particular operating system 604 or platform.

In an example embodiment, the applications 610 include a homeapplication 650, a contacts application 652, a browser application 654,a book reader application 656, a location application 658, a mediaapplication 660, a messaging application 662, a game application 664,and a broad assortment of other applications such as a third-partyapplication 666. According to some embodiments, the applications 610 areprograms that execute functions defined in the programs. Variousprogramming languages can be employed to create one or more of theapplications 610, structured in a variety of manners, such asobject-oriented programming languages (e.g., Objective-C, Java, or C++)or procedural programming languages (e.g., C or assembly language). In aspecific example, the third-party application 666 (e.g., an applicationdeveloped using the ANDROID™ or IOS™ software development kit (SDK) byan entity other than the vendor of the particular platform) may bemobile software running on a mobile operating system such as IOS™,ANDROID™, WINDOWS® Phone, or another mobile operating system. In thisexample, the third-party application 666 can invoke the API calls 612provided by the operating system 604 to facilitate functionalitydescribed herein.

Some embodiments may particularly include test automate application 667.In certain embodiments, this may be a stand-alone application thatoperates to manage communications with a server system such asthird-party servers 130 or server system 102. In other embodiments, thisfunctionality may be integrated with another application. The testautomate application 667 may request and display various data related todesigning, viewing, and updating test automates and may provide thecapability for a user 106 to input data related to the objects via atouch interface, keyboard, or using a camera device of machine 700,communication with a server system via I/O components 750, and receiptand storage of object data in memory 730. Presentation of informationand user inputs associated with the information may be managed by testautomate application 667 using different frameworks 608, library 606elements, or operating system 604 elements operating on a machine 700.

FIG. 7 is a block diagram illustrating components of a machine 700,according to some embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.Specifically, FIG. 7 shows a diagrammatic representation of the machine700 in the example form of a computer system, within which instructions716 (e.g., software, a program, an application 610, an applet, an app,or other executable code) for causing the machine 700 to perform any oneor more of the methodologies discussed herein can be executed. Inalternative embodiments, the machine 700 operates as a standalone deviceor can be coupled (e.g., networked) to other machines. In a networkeddeployment, the machine 700 may operate in the capacity of a servermachine 130, 102, 120, 122, 124, etc., or a client device 110 in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine 700 cancomprise, but not be limited to, a server computer, a client computer, apersonal computer (PC), a tablet computer, a laptop computer, a netbook,a personal digital assistant (PDA), an entertainment media system, acellular telephone, a smart phone, a mobile device, a wearable device(e.g., a smart watch), a smart home device (e.g., a smart appliance),other smart devices, a web appliance, a network router, a networkswitch, a network bridge, or any machine capable of executing theinstructions 716, sequentially or otherwise, that specify actions to betaken by the machine 700. Further, while only a single machine 700 isillustrated, the term “machine” shall also be taken to include acollection of machines 700 that individually or jointly execute theinstructions 716 to perform any one or more of the methodologiesdiscussed herein.

In various embodiments, the machine 700 comprises processors 710, memory730, and I/O components 750, which can be configured to communicate witheach other via a bus 702. In an example embodiment, the processors 710(e.g., a central processing unit (CPU), a reduced instruction setcomputing (RISC) processor, a complex instruction set computing (CISC)processor, a graphics processing unit (GPU), a digital signal processor(DSP), an application specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), another processor, or anysuitable combination thereof) include, for example, a processor 712 anda processor 714 that may execute the instructions 716. The term“processor” is intended to include multi-core processors 710 that maycomprise two or more independent processors 712, 714 (also referred toas “cores”) that can execute instructions 716 contemporaneously.Although FIG. 7 shows multiple processors 710, the machine 700 mayinclude a single processor 710 with a single core, a single processor710 with multiple cores (e.g., a multi-core processor 710), multipleprocessors 712, 714 with a single core, multiple processors 712, 714with multiples cores, or any combination thereof.

The memory 730 comprises a main memory 732, a static memory 734, and astorage unit 736 accessible to the processors 710 via the bus 702,according to some embodiments. The storage unit 736 can include amachine-readable medium 738 on which are stored the instructions 716embodying any one or more of the methodologies or functions describedherein. The instructions 716 can also reside, completely or at leastpartially, within the main memory 732, within the static memory 734,within at least one of the processors 710 (e.g., within the processor'scache memory), or any suitable combination thereof, during executionthereof by the machine 700. Accordingly, in various embodiments, themain memory 732, the static memory 734, and the processors 710 areconsidered machine-readable media 738.

As used herein, the term “memory” refers to a machine-readable medium738 able to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While themachine-readable medium 738 is shown, in an example embodiment, to be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storethe instructions 716. The term “machine-readable medium” shall also betaken to include any medium, or combination of multiple media, that iscapable of storing instructions (e.g., instructions 716) for executionby a machine (e.g., machine 700), such that the instructions 716, whenexecuted by one or more processors of the machine 700 (e.g., processors710), cause the machine 700 to perform any one or more of themethodologies described herein. Accordingly, a “machine-readable medium”refers to a single storage apparatus or device, as well as “cloud-based”storage systems or storage networks that include multiple storageapparatus or devices. The term “machine-readable medium” shallaccordingly be taken to include, but not be limited to, one or more datarepositories in the form of a solid-state memory (e.g., flash memory),an optical medium, a magnetic medium, other non-volatile memory (e.g.,erasable programmable read-only memory (EPROM)), or any suitablecombination thereof. The term “machine-readable medium” specificallyexcludes non-statutory signals per se.

The I/O components 750 include a wide variety of components to receiveinput, provide output, produce output, transmit information, exchangeinformation, capture measurements, and so on. In general, it will beappreciated that the I/O components 750 can include many othercomponents that are not shown in FIG. 7. The I/O components 750 aregrouped according to functionality merely for simplifying the followingdiscussion, and the grouping is in no way limiting. In various exampleembodiments, the I/O components 750 include output components 752 andinput components 754. The output components 752 include visualcomponents (e.g., a display such as a plasma display panel (PDP), alight emitting diode (LED) display, a liquid crystal display (LCD), aprojector, or a cathode ray tube (CRT)), acoustic components (e.g.,speakers), haptic components (e.g., a vibratory motor), other signalgenerators, and so forth. The input components 754 include alphanumericinput components (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point-based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or other pointinginstruments), tactile input components (e.g., a physical button, a touchscreen that provides location and force of touches or touch gestures, orother tactile input components), audio input components (e.g., amicrophone), and the like.

In some further example embodiments, the I/O components 750 includebiometric components 756, motion components 758, environmentalcomponents 760, or position components 762, among a wide array of othercomponents. For example, the biometric components 756 include componentsto detect expressions (e.g., hand expressions, facial expressions, vocalexpressions, body gestures, or eye tracking), measure biosignals (e.g.,blood pressure, heart rate, body temperature, perspiration, or brainwaves), identify a person (e.g., voice identification, retinalidentification, facial identification, fingerprint identification, orelectroencephalogram based identification), and the like. The motioncomponents 758 include acceleration sensor components (e.g.,accelerometer), gravitation sensor components, rotation sensorcomponents (e.g., gyroscope), and so forth. The environmental components760 include, for example, illumination sensor components (e.g.,photometer), temperature sensor components (e.g., one or morethermometers that detect ambient temperature), humidity sensorcomponents, pressure sensor components (e.g., barometer), acousticsensor components (e.g., one or more microphones that detect backgroundnoise), proximity sensor components (e.g., infrared sensors that detectnearby objects), gas sensor components (e.g., machine olfactiondetection sensors, gas detection sensors to detect concentrations ofhazardous gases for safety or to measure pollutants in the atmosphere),or other components that may provide indications, measurements, orsignals corresponding to a surrounding physical environment. Theposition components 762 include location sensor components (e.g., aGlobal Positioning System (GPS) receiver component), altitude sensorcomponents (e.g., altimeters or barometers that detect air pressure fromwhich altitude may be derived), orientation sensor components (e.g.,magnetometers), and the like.

Communication can be implemented using a wide variety of technologies.The I/O components 750 may include communication components 764 operableto couple the machine 700 to a network 780 or devices 770 via a coupling782 and a coupling 772, respectively. For example, the communicationcomponents 764 include a network interface component or another suitabledevice to interface with the network 780. In further examples,communication components 764 include wired communication components,wireless communication components, cellular communication components,near field communication (NFC) components, BLUETOOTH® components (e.g.,BLUETOOTH® Low Energy), WI-FI® components, and other communicationcomponents to provide communication via other modalities. The devices770 may be another machine 700 or any of a wide variety of peripheraldevices (e.g., a peripheral device coupled via a Universal Serial Bus(USB)).

Moreover, in some embodiments, the communication components 764 detectidentifiers or include components operable to detect identifiers. Forexample, the communication components 764 include radio frequencyidentification (RFID) tag reader components, NFC smart tag detectioncomponents, optical reader components (e.g., an optical sensor to detecta one-dimensional bar codes such as a Universal Product Code (UPC) barcode, multi-dimensional bar codes such as a Quick Response (QR) code,Aztec Code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code,Uniform Commercial Code Reduced Space Symbology (UCC RSS)-2D bar codes,and other optical codes), acoustic detection components (e.g.,microphones to identify tagged audio signals), or any suitablecombination thereof. In addition, a variety of information can bederived via the communication components 764, such as location viaInternet Protocol (IP) geo-location, location via WI-FI® signaltriangulation, location via detecting a BLUETOOTH® or NFC beacon signalthat may indicate a particular location, and so forth.

In various example embodiments, one or more portions of the network 780can be an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), the Internet, a portion of the Internet, a portion of the publicswitched telephone network (PSTN), a plain old telephone service (POTS)network, a cellular telephone network, a wireless network, a WI-FI®network, another type of network, or a combination of two or more suchnetworks. For example, the network 780 or a portion of the network 780may include a wireless or cellular network, and the coupling 782 may bea Code Division Multiple Access (CDMA) connection, a Global System forMobile communications (GSM) connection, or another type of cellular orwireless coupling. In this example, the coupling 782 can implement anyof a variety of types of data transfer technology, such as SingleCarrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized(EVDO) technology, General Packet Radio Service (GPRS) technology,Enhanced Data rates for GSM Evolution (EDGE) technology, thirdGeneration Partnership Project (3GPP) including 3G, fourth generationwireless (4G) networks, Universal Mobile Telecommunications System(UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability forMicrowave Access (WiMAX), Long Term Evolution (LTE) standard, othersdefined by various standard-setting organizations, other long rangeprotocols, or other data transfer technology.

In example embodiments, the instructions 716 are transmitted or receivedover the network 780 using a transmission medium via a network interfacedevice (e.g., a network interface component included in thecommunication components 764) and utilizing any one of a number ofwell-known transfer protocols (e.g., Hypertext Transfer Protocol(HTTP)). Similarly, in other example embodiments, the instructions 716are transmitted or received using a transmission medium via the coupling772 (e.g., a peer-to-peer coupling) to the devices 770. The term“transmission medium” shall be taken to include any intangible mediumthat is capable of storing, encoding, or carrying the instructions 716for execution by the machine 700, and includes digital or analogcommunications signals or other intangible media to facilitatecommunication of such software.

Furthermore, the machine-readable medium 738 is non-transitory (in otherwords, not having any transitory signals) in that it does not embody apropagating signal. However, labeling the machine-readable medium 738“non-transitory” should not be construed to mean that the medium isincapable of movement, the medium 738 should be considered as beingtransportable from one physical location to another. Additionally, sincethe machine-readable medium 738 is tangible, the medium 738 may beconsidered to be a machine-readable device.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Although an overview of the inventive subject matter has been describedwith reference to specific example embodiments, various modificationsand changes may be made to these embodiments without departing from thebroader scope of embodiments of the present disclosure

The embodiments illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other embodiments may be used and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. The Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Moreover, plural instances may be provided forresources, operations, or structures described herein as a singleinstance. Additionally, boundaries between various resources,operations, modules, engines, and data stores are somewhat arbitrary,and particular operations are illustrated in a context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within a scope of various embodiments of thepresent disclosure. In general, structures and functionality presentedas separate resources in the example configurations may be implementedas a combined structure or resource. Similarly, structures andfunctionality presented as a single resource may be implemented asseparate resources. These and other variations, modifications,additions, and improvements fall within a scope of embodiments of thepresent disclosure as represented by the appended claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A computer-implemented method comprising: foreach of a plurality of test automates, determining whether a first testautomate is associated with a same application as a second test automatebased on a similarity score associated with actions in common betweenthe first test automate and the second test automate, by: determiningthat an application identifier associated with the first test automateis not the same as an application identifier of the second testautomate; and comparing the actions of the first test automate with theactions of the second test automate to determine the similarity score;based on determining that the first test automate is associated with thesame application as the second test automate, for each action in thefirst test automate: determining a common action in the second testautomate; determining whether an element identifier in the first testautomate for the common action has changed in the second test automate;and based on determining that an element identifier in the first testautomate for the common action has changed in the second test automate,recording the changed element identifier in a data store, the data storecomprising a list of changed element identifiers for each of a pluralityof applications; and providing the data store comprising the list ofchanged element identifiers for each of the plurality of applications toa computing device, wherein element identifiers in at least one customtest automate corresponding to the changed element identifiers for atleast one application are automatically determined based on the list ofchanged element identifiers for each of the plurality of applications.2. The method of claim 1, wherein determining whether the first testautomate is associated with the same application as the second testautomate is based on comparing the application identifier associatedwith the first test automate to the application identifier of the secondtest automate to confirm that they are the same.
 3. The method of claim1, wherein the application identifier associated with the first testautomate or the second test automate is a Uniform Resource Locator or aUniform Resource Identifier.
 4. The method of claim 1, whereindetermining whether the first test automate is associated with the sameapplication as the second test automate based on a similarity of actionsin common between the first test automate and the second test automatefurther comprises: determining whether the similarity score meets asimilarity threshold; and determining that the first test automate isassociated with the same application as the second test automate basedon determining that the similarity score meets the similarity threshold.5. The method of claim 4, wherein the similarity score is based on apercentage of similarity between the actions of the first test automateand the actions of the second test automate.
 6. The method of claim 4,further comprising: recording the application identifier of the secondautomate in the data store as a changed application identifier.
 7. Themethod of claim 1, wherein determining whether the first test automateis associated with the same application as the second test automate isbased on the application identifier associated with the first testautomate and the application identifier associated with the second testautomate and based on a similarity of actions in common between thefirst test automate and the second test automate.
 8. The method of claim1, wherein automatically determining element identifiers in at least onecustom test automate corresponding to the changed element identifiersfor at least one application based on the list of changed elementidentifiers for each of the plurality of applications comprises: foreach custom test automate, determining an application identifierassociated with an application for the custom test automate; determiningelement identifiers in the custom test automate; accessing the datastore to determine whether any element identifiers in the custom testautomate associated with the application identifier have been changed;and updating the element identifiers in the custom test automate thathave been changed with new element identifiers identified in the datastore.
 9. The method of claim 8, further comprising: after determiningan application identifier associated with an application for a thirdtest automate, determining that the application identifier does notcorrespond to an application identifier in the data store; comparing theactions of the third test automate with the actions of each of aplurality of standard test automates in the data score to determine asimilarity score; determining whether the similarity score meets asimilarity threshold; and determining that the third test automate isassociated with a same application as a fourth test automate based ondetermining that the similarity score meets the similarity threshold.10. A system comprising: a memory that stores actions; and one or moreprocessors configured by the actions to perform operations comprising:for each of a plurality of test automates, determining whether a firsttest automate is associated with a same application as a second testautomate based on a similarity score associated with actions in commonbetween the first test automate and the second test automate, by:determining that an application identifier associated with the firsttest automate is not the same as an application identifier of the secondtest automate; and comparing the actions of the first test automate withthe actions of the second test automate to determine the similarityscore; based on determining that the first test automate is associatedwith the same application as the second test automate, for each actionin the first test automate: determining a common action in the secondtest automate; determining whether an element identifier in the firsttest automate for the common action has changed in the second testautomate; and based on determining that an element identifier in thefirst test automate for the common action has changed in the second testautomate, recording the changed element identifier in a data store, thedata store comprising a list of changed element identifiers for each ofa plurality of applications; and providing the data store comprising thelist of changed element identifiers for each of the plurality ofapplications to a computing device, wherein element identifiers in atleast one custom test automate corresponding to the changed elementidentifiers for at least one application are automatically determinedbased on the list of changed element identifiers for each of theplurality of applications.
 11. The system of claim 10, whereindetermining whether the first test automate is associated with the sameapplication as the second test automate is based on comparing theapplication identifier associated with the first test automate to theapplication identifier of the second test automate to confirm that theyare the same.
 12. The system of claim 10, wherein the applicationidentifier associated with the first test automate or the second testautomate is a Uniform Resource Locator or a Uniform Resource Identifier.13. The system of claim 10, wherein determining whether the first testautomate is associated with the same application as the second testautomate based on a similarity of actions in common between the firsttest automate and the second test automate further comprises:determining whether the similarity score meets a similarity threshold;and determining that the first test automate is associated with the sameapplication as the second test automate based on determining that thesimilarity score meets the similarity threshold.
 14. The system of claim13, wherein the similarity score is based on a percentage of similaritybetween the actions of the first test automate and the actions of thesecond test automate.
 15. The system of claim 12, the operations furthercomprising: recording the application identifier of the second automatein the data store as a changed application identifier.
 16. The system ofclaim 10, wherein determining whether the first test automate isassociated with the same application as the second test automate isbased on the application identifier associated with the first testautomate and the application identifier associated with the second testautomate and based on a similarity of actions in common between thefirst test automate and the second test automate.
 17. The system ofclaim 10, wherein automatically determining element identifiers in atleast one custom test automate corresponding to the changed elementidentifiers for at least one application based on the list of changedelement identifiers for each of the plurality of applications comprises:for each custom test automate, determining an application identifierassociated with an application for the custom test automate; determiningelement identifiers in the custom test automate; accessing the datastore to determine whether any element identifiers in the custom testautomate associated with the application identifier have been changed;and updating the element identifiers in the custom test automate thathave been changed with new element identifiers identified in the datastore.
 18. The system of claim 17, further comprising: after determiningan application identifier associated with an application for a thirdtest automate, determining that the application identifier does notcorrespond to an application identifier in the data store; comparing theactions of the third test automate with the actions of each of aplurality of standard test automates in the data score to determine asimilarity score; determining whether the similarity score meets asimilarity threshold; and determining that the third test automate isassociated with a same application as a fourth test automate based ondetermining that the similarity score meets the similarity threshold.19. A non-transitory computer-readable medium comprising actions storedthereon that are executable by at least one processor to cause acomputing device to perform operations comprising: for each of aplurality of test automates, determining whether a first test automateis associated with a same application as a second test automate based ona similarity score associated with actions in common between the firsttest automate and the second test automate, by: determining that anapplication identifier associated with the first test automate is notthe same as an application identifier of the second test automate; andcomparing the actions of the first test automate with the actions of thesecond test automate to determine the similarity score; based ondetermining that the first test automate is associated with the sameapplication as the second test automate, for each action in the firsttest automate: determining a common action in the second test automate;determining whether an element identifier in the first test automate forthe common action has changed in the second test automate; and based ondetermining that an element identifier in the first test automate forthe common action has changed in the second test automate, recording thechanged element identifier in a data store, the data store comprising alist of changed element identifiers for each of a plurality ofapplications; and providing the data store comprising the list ofchanged element identifiers for each of the plurality of applications toa computing device, wherein element identifiers in at least one customtest automate corresponding to the changed element identifiers for atleast one application are automatically determined based on the list ofchanged element identifiers for each of the plurality of applications.20. The non-transitory computer-readable medium of claim 19, whereindetermining whether the first test automate is associated with the sameapplication as the second test automate based on a similarity of actionsin common between the first test automate and the second test automatefurther comprises: determining whether the similarity score meets asimilarity threshold; and determining that the first test automate isassociated with the same application as the second test automate basedon determining that the similarity score meets the similarity threshold.